Multimodal input processing

ABSTRACT

Exemplary embodiments described herein provide methods, mediums, and systems for recognizing a formalism type present in an input and generating an output that is consistent with the formalism type. In order to generate an output that is consistent with the formalism type, exemplary embodiments analyze elements of the input together. Such a holistic analysis determines or uses a likelihood that the different elements coexist together in a given formalism type. Based on this holistic analysis, an appropriate formalism type that is consistent with the coexistence of the elements may be selected. After the appropriate formalism type is selected, the input may be processed to generate an output consistent with the formalism type. The output may be computer-based representations of the input defined according to a program associated with the identified formalism type, and/or the input may be translated or transformed into another representation.

DESCRIPTION OF THE DRAWINGS

FIGS. 1A-1B depict an example of identifying a formalism in an input andgenerating an output corresponding to the formalism.

FIG. 2 depicts an example of an input including structures representedaccording to formalisms.

FIG. 3 is a flowchart describing an exemplary preprocessing technique tobe performed on an input to identify structures in the input.

FIG. 4A is an exemplary probability table used to determine a likelihoodthat two or more elements coexist in a given formalism based on ananalysis of elements in an input image.

FIG. 4B is an exemplary probability table used to determine a likelihoodthat an image is associated with a given formalism based on a holisticanalysis of the image.

FIG. 5 is an exemplary model for selecting a formalism based on alikelihood that two or more elements coexist in a given formalism.

FIGS. 6A-6C describe exemplary formalism-specific processing techniques.

FIGS. 7-13 depict exemplary outputs, transformations, and translationsof different types of inputs.

FIG. 14 depicts an electronic device suitable for use with exemplaryembodiments.

FIG. 15 depicts an exemplary network configuration suitable for use withexemplary embodiments.

DETAILED DESCRIPTION

Exemplary embodiments described herein provide methods, mediums, andsystems for recognizing a formalism type present in an input andgenerating an output that is consistent with the formalism type. Theoutput may be generated using formalism-specific processing techniquesthat are triggered by the recognition that the input includes aparticular formalism type.

By providing the capability to recognize formalism types in an input andprocess the input in a formalism-dependent manner, many different typesof inputs can be translated into computer-based representations. Forexample, a teacher may draw a block diagram on a whiteboard, and astudent may snap a photograph of the block diagram which may then betranslated into an operable (e.g., having executable semantics) model inblock diagramming software. In another example, a user may sketch amathematical equation, or take a picture of an equation in a book, andthe equation may be translated into computer code or a reference to acomputer-executable function that implements the equation.

A formalism may represent a way to organize or represent information. Aformalism may define or describe a syntax for structuring theinformation and/or a valid alphabet for representing the information.Different formalisms may be identified by different notation styles. Insome cases, formalisms may also be known as modalities or domains.Examples of formalisms include, but are not limited to, a table, a blockdiagram model, a statechart, a Unified Modeling Language (UML) diagram,a mathematical equation, a chemical structure, a chart, and/or, a plot.

According to exemplary embodiments, inputs (e.g., an input to a model)may include one or more types of formalisms. The input may be identifiedusing exemplary embodiments and a structure of the input determined. Forexample, a structure of an input structure may be made up of two or morerelated elements (e.g., squares, circles, lines, arrows, text, etc.)that are arranged according to a formalism type. The formalism type ofthe structure may be unknown to a system processing the input at thetime the input is accessed.

In order to generate an output that is consistent with the unknownformalism type, exemplary embodiments analyze elements of the structurein context. For example, if an input includes two elements, exemplaryembodiments may analyze the elements together. The analysis maydetermine or use a likelihood that the different elements coexisttogether in a given formalism type. Based on this analysis, anappropriate formalism type that is consistent with the coexistence ofthe elements may be selected (e.g., from a library that includes aquantity of valid formalism types).

After the appropriate formalism type is selected, the input may beprocessed to generate an output consistent with the formalism type. Inexemplary embodiments, the output may include one or more computer-basedrepresentations of the input defined according to a program associatedwith the identified formalism type. Alternatively, or in addition theinput may be translated or transformed into another representation. Forexample, an input equation may be transformed into computer coderepresenting the equation, or a table of data may be transformed into agraph of the data.

An example of a process for converting an input of an unknown formalismtype into an output is shown in FIGS. 1A-1B.

In the top-most panel of FIG. 1A, an exemplary input image is depicted.In this example, the image is of a hand-drawn block diagram model. Theblock diagram model includes a sine block and a cosine block providingoutputs to a sum block, where the output of the sum block is provided toa scope block. Several steps may be performed in order to identify theformalism (block diagram model) in the image, and to generate acomputer-based representation (e.g., a Simulink model) of the input.

The middle panel of FIG. 1A depicts an exemplary preprocessing stepperformed using a computing device. The preprocessing step identifiesone or more structures in the image. For example, the entire blockdiagram model may be a single structure made up of interconnectedstructures representing the blocks of the block diagram model. Theinterconnected structures making up the block diagram model may bereferred to as elements and may include, but are not limited to,boxes/blocks, arrows, and text. In order to arrive at the representationshown in the middle panel of FIG. 1A, the lines representing the boxesand arrows may be identified and isolated (e.g., by using anedge-detection algorithm), and the text in the boxes may be extracted(e.g., using an optical character recognition algorithm).

After the block diagram model elements are isolated, the elements may beanalyzed together to determine a likelihood that the elements wouldcoexist in a given formalism (e.g., an executable time-based blockdiagram model). Analyzing the elements together allows the formalism toorganize the input structure(s) to be identified. This may beaccomplished, for example, by consulting a table of probabilities. Anexample of such a table is depicted in the bottom panel of FIG. 1A. Thetable of probabilities may be a computer-readable data store stored in anon-transitory computer readable medium (e.g., random access memory).

This exemplary table includes a list of possible elements that mayappear in a structure (along the left-most column), and a list ofpossible formalisms (along the top-most row). Each cell in the tablerepresents a probability that the image includes the formalism in thetop-most row, given that the structure in the left-most column isidentified in the image. These probabilities may be determined, forexample, by analyzing samples of the formalisms to identify aprobability that different elements occur in the formalism.

For example, as shown in the exemplary probability table of FIG. 1A, theinput image could contain elements organized according to a Tableformalism or according to a Block Diagram formalism (among otherpossibilities). In other words, the elements could come together to makea table, or could come together to make a block diagram.

According to the probability table of FIG. 1A, the probability of therebeing a Table formalism in the image (given that a box element ispresent in the image) is determined to be 33%, whereas the probabilityof that there is a Block Diagram formalism in the image (given that abox element is present in the image) is determined to be 30%. If the boxelement were analyzed in isolation, there is a greater likelihood that atable formalism is present in the image (33% likelihood) rather than ablock diagram (30% likelihood).

Exemplary embodiments reduce the chance of misclassification byexamining the elements together in order to determine a likelihood thatthe elements would coexist in a given formalism. In this case, the tablealso includes several arrow elements. As shown in the table, theprobability of seeing an arrow element in the Table formalism isrelatively low (5%) while the probability of seeing an arrow element inthe Block Diagram formalism is relatively high (40%).

In one embodiment, the individual probabilities that each element isfound in a given formalism may be added together in order to generate anoverall score for the formalism. In the example described above, theoverall score for the table formalism is 38 (33%+5%), while the overallscore for the block diagram formalism is 70 (30%+40%). The block diagramformalism may be selected since it has the highest score.

Exemplary embodiments can use the probabilities from the table toidentify and select a formalism type that is compatible with thecoexistence of elements existing in an input.

Referring now to FIG. 1B, exemplary embodiments may select a processingtechnique that is specific to the selected formalism type. For example,the computed scores from the table of FIG. 1A may be used to select aprocessing technique such as the one shown at the top of FIG. 1B. Theflowchart in the top panel of FIG. 1B converts elements identified inthe pre-processing step (e.g., a square with the word “cos” in themiddle) into formalism-specific elements (e.g., a cosine block in ablock diagram model). The details of this particular extraction methodwill be discussed in more detail below with respect to the processing ofa block diagram model.

After performing the formalism-specific processing, the result may be anoutput that is represented in a manner consistent with the formalismidentified in the classification step in the bottom panel of FIG. 1A. Inthe example of FIGS. 1A-1B, the output is a computer-based block diagrammodel such as the one depicted in the bottom panel of FIG. 1B. Thecomputer-based block diagram model may be simulated or executed to carryout the functionality of the model that was originally drawn in theinput. In this case, the output is imbued with functionality, such asgenerating a sine and cosine signal, adding the signals, and displayingthe results. The output blocks may be made up of, or associated with,executable code that, when executed, performs the functions labeled onthe blocks.

Each of the above-described steps (e.g., receiving the input,pre-processing, classification, formalism-specific extraction, andgenerating an output) will now be described in more detail.

Initially, an input may be accessed or created. The input may include,for example, an image (e.g., a digital photograph or scanned image),audio data, gestural data, haptic data (e.g., information about how harda user presses on an input device, or how much surface area the user'sfinger presents to a touch screen), interaction data (e.g., live datarecorded from an interaction with a whiteboard), printouts, etc. Inputmay also be hierarchical, in that one type of input may include othertypes of inputs. Examples of such hierarchical input may include videodata (from which images, sounds, and/or gestural data may be retrieved),etc.

FIG. 2 depicts an example of an input 200. Input 200 is a scanned imageof a hand-drawn block diagram 218 with an embedded state diagram 220.

Input 200 may contain two or more elements. Elements may representfeatures (e.g., predetermined shapes), strings of text, or other basicparts that represent valid building blocks that may be present in aninput according to a formalism.

In the input 200 of FIG. 2, several elements 202-214 (among others notspecifically referenced) are present. For example, the input 200includes a first circle element 202, a first straight arrow element 204,a first box element 206, a second straight arrow element 208, a secondcircle element 210, a curved arrow element 212, and a second box element214.

The elements may be organized into structures such as block diagram 218and state diagram 220. A structure may be used to represent groups ofrelated elements that are organized according to a common formalism type(e.g., time-based, event-based, state-based, dataflow-based). An inputmay include more than one structure, and different structures may berepresented according to different formalism types. Thus, multipleformalism types may be present in an input.

The different structures and/or formalism types in an input 200 may bedistinguished in a variety of ways. For example, in the input 200, thestate diagram 220 is provided inside of the second box element 214,which is designated by dashed lines. The presence of a structure insideof a dashed box may be used to indicate that the structure in the dashedbox is distinct from structures outside the dashed box.

Different structures may also be distinguished using different visual oraudio characteristics. For example, the state diagram 220 could havebeen drawn using a different color than the block diagram 218, or usinga different line thickness, or (in the case of an audio input) usingdifferent terms for the elements in the input (e.g., referring to a boxas a “block” when discussing a time-based block diagram, as opposed to a“state” in a state diagram) or different pitches to describe thedifferent structures. Alternatively or in addition, the differentstructures may be set apart using special characters, keywords,gestures, or other indicators.

If visual or audio characteristics are regularly used to distinguishdifferent structures, then a meaning of the visual or audiocharacteristics may be learned by the system processing the input. Thelearned meaning of the visual or audio characteristics may be applied tofuture inputs (or to different structures within the same input). Othertypes of machine learning may also be leveraged.

The input may contain information related to parameters or metainformation of the model or model elements. For example, the input mayinclude execution information (e.g., which solver to use for executing amodel, what step size to use, what tolerances to use, whether to storemodel output values), code generation information (e.g., whichoptimizations to apply to code generation, whether to generate HDL or C,which performance characteristics such as speed, size, power, tooptimize for), etc. Additionally or alternatively, the input may provideblock parameter information (e.g., initial state values, transferfunction parameter values, addition or subtraction operations for portsof summation block, data types of input/output/internal variables).Furthermore, information about reuse of diagram fragments (e.g., bycreating library blocks) may be indicated graphically (e.g., by labelinga diagram fragment and referencing this label elsewhere on the diagram),and information may be provided about signals (e.g., their name, whethertheir signal values should be logged, what variable to use to store thedata, what dimension the signal has, etc.).

The input may contain information related to editing, navigation, andexecution operations. For example, a particular input may cause thediagram to be processed and converted into a data structure. Anotherinput may cause the processed diagram to be executed. Furthermore,results may be projected back onto the input drawing or converteddrawing. Such results may include simulation output and/or animationinformation (such as indicating active states in a state transitiondiagram during execution). Still further, the input may indicate that aparticular element may be copied elsewhere, deleted, converted into adifferent element, etc. During execution, input may be provided thatpauses or resumes execution. Moreover, input may be provided thatindicates where execution breakpoints (e.g., for debugging) should beset.

Although the example input 200 in FIG. 2 depicts two differentstructures belonging to two different formalism types, multipledifferent structures having the same formalism type may also be presentin the input 200. For example, an input may include two time-based blockdiagrams drawn by a user.

Different structures organized according to different formalism typesmay be arranged hierarchically. For example, in the input 200, the statediagram 220 is intended to be contained within the first box element 206of the block diagram model 218.

Hierarchies may be designated in a number of ways. In FIG. 2, ahierarchy is indicated by pairing the second straight arrow element 208with the second box element 214 (where the second box element 214 isrepresented with dashed lines). The presence of an arrow and a dashedbox may be used to indicate that the structure(s) present inside thedashed box are intended to be nested at a second hierarchical layerinside of an element to which the arrow is pointing.

Hierarchies may also be represented using other visual, audio, orgestural designations. For example, predetermined colors, keywords,sounds, or gestures may be used to signify a hierarchy and/or indicate arelative arrangement of structures in the hierarchy.

After an input is received, the input may be preprocessed in order toidentify the structures in the input and/or to identify the elementsthat make up the structures. FIG. 3 is a flowchart depicting an exampleof a preprocessing technique 300.

Optionally, at step 302 the image may be normalized to facilitateprocessing. For example, portions of the image may be scaled up or downin size to match predetermined reference sizes (e.g., from a library ofelements, such as the elements included in the above-describedprobability table) in order to assist in the identification orrecognition of shapes, text, or other features in the image.

At step 304, an edge detection algorithm may be carried out. The edgedetection algorithm may identify, for example, lines in the image. Thelines may be identified by comparing color values in the image andidentifying a line when the color changes by more than a predeterminedthreshold tolerance. Alternatively or in addition, contrast or intensityof pixels in the image may be compared to determine the locations oflines in the image. If one or more lines are connected, such as intopredetermined shapes like boxes, circles, or triangles, the shapes maybe identified.

At step 306, the lines identified at step 304 may be used to determinethe elements present in the image. For example, a database ofpredetermined valid features may be stored and compared to the lines ofthe image. If one of the predetermined valid features is found in theimage, this may be flagged as a particular type of element in the image.

For example, some of the lines identified at step 304 may be connectedinto a box shape (e.g., substantially four lines connected atsubstantially ninety-degree angles). The database of predetermined validfeatures may include a shape corresponding to a box. If the elements inthe image are normalized at step 302, the size of the box in the imagemay generally correspond to the size of the box in the database ofpredetermined valid features. If the box in the image corresponds to thebox in the database (e.g., there is a similarity between the box in theinput image and the reference box from the database that is more than apredetermined threshold amount), then the system may identify a boxelement in the input image.

In some circumstances, the identity of an element in the image may beambiguous. For example, a user may draw a rectangle that has roundedcorners. It may not be readily apparent whether the user intended todraw a rectangle, a circle, or a rounded rectangle. In exemplaryembodiments, context information about the formalism in which theelement is found may be used to assist in interpreting the user'sintentions. For example, if other elements in the input suggest that theformalism is a statechart, then the object may be interpreted as beingmore likely to be a circle (e.g., to represent a transition junction) ora rectangle with rounded corners (e.g., to represent a state). If theother elements suggest that the formalism is a table, then the objectmay be interpreted as being more likely to be a rectangle, which issometimes used to represents cells in a table.

Accordingly, the image may be analyzed cumulatively. For example, a“running tally” of likelihoods for each formalism type may be calculatedas the image is processed. The identity of the element may beinterpreted based on the running tally and the current “most-likely”formalism candidate. Alternatively or in addition, processing mayproceed in two or more passes. In the first pass, unambiguous elementsmay be identified or elements may be identified with a high probabilityand used to make an initial prediction of the formalism involved.Ambiguous elements or elements with a lower probability may then beprocessed in a second pass, based on the initial prediction of theformalism.

In some embodiments, the elements in the image may be unnormalized. Forexample, the system may identify a shape in the image, and may determinea proximity and/or relationship of the identified shape to other shapesin the image (e.g., a square connected to a circle by a line having anarrow). The identified shapes, proximities, and relationships may beused to determine which elements are present (or most likely present) inthe image.

After the elements making up the structures are identified, the elementsmay be analyzed together in order to evaluate a likelihood that theelements would coexist in a given formalism. The evaluation may involve,for example, consulting a table of probabilities. FIGS. 4A and 4Binclude exemplary tables of probabilities 400 and 420, respectively.

The table of probabilities 400 is an exemplary table analyzing thestructures present in the input. The table of probabilities 400 mayinclude a row of formalism types 402. The row of formalism types 402 maycorrespond to a list of valid formalisms stored in a data store, such asa formalism library. The list of valid formalisms may be a predeterminedlist that includes any formalism for which a formalism-specificprocessing technique is available. The list of valid formalisms may beextended by adding additional formalism-specific processing techniques,and by specifying any elements that may occur in the formalism.

The table of probabilities 400 may include a column of valid elements404. The valid elements 404 may be elements that are expected to beencountered in the formalism types 402. Cells in the column of validelements 404 may include one or more valid elements 406. Valid elements406 may be added, for example when a new formalism is added (or when newelements are added to an existing formalism). Valid elements 406 may becompared against elements identified in the input.

In some embodiments, the valid elements 406 may be used to identify theelements in the input. In these embodiments, the elements in the inputmay be resized in step 302 to a standard size that corresponds to thetypes of valid elements 406 in the probability table 400. The types ofvalid elements 406 may then be compared to the elements in the inputimage, and a nearest match to the element in the image may be selected.

The table of probabilities 400 may include a number of cells 408. Thecells 408 may define, at the intersection of a particular formalism typein the row of formalism types 402 and element type in the column ofelement types 404, a probability that the element type appears in theformalism represented by the formalism type. Alternatively or inaddition, the table may include contextual information and/or holisticinformation about the image. For example, FIG. 4B depicts a table ofprobabilities 420 that includes contextual and holistic information. Thecontextual and/or holistic information may be used to determine theformalism(s) present in the image.

In the table 420, the element type 410 represents a situation in whichtext or symbols coexist with a block in an input, and the text/symbolsexist inside the block. As shown in FIG. 4, this situation may indicatea relatively high likelihood that the formalism type is a table, blockdiagram, or UML diagram. On the other hand, if the text/symbols occuroutside of the context of a block (either outside of a block in theinput, or in an input in which no block exists), this may indicate ahigh likelihood that the formalism type is a plot or equation.

In another example of holistic analysis, the element type 412 representsa dimensionality reduction of the image using robust principal componentanalysis (PCA). In a PCA, the image is compared to a predefined libraryof templates. A feature space may be calculated for the templates in thepredefined library, and the input image may also be reduced to a featurespace. A Euclidean distance between the feature space(s) of one or moretemplates and the feature space of the input image may be calculated.These Euclidean distances may be normalized to yield probabilities forthe formalisms represented by the templates.

The column of element types 404 may further include image patchfeatures. For example, the element type 414 represents a histogram oforiented gradient features. A Histogram of Oriented Gradients (HoG)describes the directionality of edges (intensity differences ofadjoining pixels) in an image patch. With prior training, an applicationcan classify an image patch based on its HoG features. For example, animage patch containing text or equations will have a different featuredescription than an image patch containing only all or part of a blockdiagram. HoG is one of the features that can be used to perform thiskind of image content classification for a patch.

In another type of analysis, directionality information may be used tointerpret elements in the input. For example, if a user draws a linefrom left-to-right connecting two rectangles, the line may beinterpreted as a signal line in a block diagram model (typicallyrepresented by an arrow showing the output to input of informationbetween two blocks). The directionality of the line (left-to-right) inthis case may be interpreted to provide a direction for the informationtransfer (e.g., the block on the left may be identified as the source ofthe information, while the block on the right may be identified as therecipient of the information). An arrow may be created in the blockdiagram model generated from the input with the appropriatedirectionality.

A holistic analysis (as shown in the table 420) may be used separatelyor in combination with a structural analysis (as shown in the table400). In some cases, the same implementation may be represented eitherstructurally or based on holistic analysis. For example, if multipleblocks are present in the image, and the blocks are arrangedside-by-side in close proximity, then there may be an increasedlikelihood that the formalism in the image is a table. Thisimplementation may be represented by creating a single structureincluding multiple blocks side-by-side (as shown by the sixth element inthe table of FIG. 4A), or by including a holistic element in the table420 querying whether the blocks are within a predetermined distance ofeach other.

The probabilities in the tables of probabilities 400 and 420 may bepredetermined based on, for example, analyses of samples of structuresknown to be in the formalism, or may be assigned based on expertknowledge of the formalism. The probabilities may be modified dependingon the application. The probabilities may be static or may bedynamically updated. For example, as more formalisms of a given type areencountered, the probability assigned to each cell may be updated toreflect the probabilities of each formalism/structure pair based on therate that each formalism/structure pair is encountered in operation.

In some embodiments, the probabilities may be adaptively updated basedon feedback, such as user feedback. For example, a computing system maymake an initial prediction of what formalism a given input is associatedwith, and a user may provide feedback as to whether the prediction iscorrect or incorrect. If the prediction is incorrect, the user mayfurther indicate what the correct formalism is. Based on this feedback,the computing system may update the table of probabilities to raise orlower the probability of one or more structures in one or moreformalisms. For example, the computing system may lower theprobabilities assigned to the structures in the input for the initial(incorrect) predicted formalism, and/or may raise the probabilitiesassigned to the structures in the input for the correct formalism.

It is noted that the specific probabilities depicted in FIGS. 4A and 4Bare intended to be exemplary.

The probabilities in the table of probabilities 400 may be used asvariables of a model used to select a formalism type. An example of sucha model 500, in the form of a probabilistic undirected graph is shown inFIG. 5.

The model 500 may include a first layer of nodes 502 corresponding toeach of the elements in a structure. A value x_(i) of each node in thefirst layer 502 may indicate, for example a number of the elementscorresponding to the node in the structure. Alternatively, the valuex_(i) may be a binary value (e.g., x_(i)=0 if none of the elementscorresponding to where x_(i) are in the structure, or x_(i)=1 is thestructure contains at least one element corresponding to x_(i)).

The model 500 may further include a second layer of nodes 504representing a set of costs or weights assigned to different formalismsg_(i). The cost or weights may correspond to the probabilities of theprobability table 400 summed together for a given formalism g_(i) togenerate an overall score for the formalism. For example, if the inputimage included an arrow and a diamond, then the overall score for theUML Diagram formalism (refer to FIG. 4) may be 1.0 (0.2+0.8). Theoverall score for the Table formalism, on the other hand, may be 0.06(0.05+0.01).

The model 500 may include a processing block 506 for selecting thehighest-scoring formalism from among the formalisms g_(i). In the aboveexample, the UML Diagram formalism may be selected over the Tableformalism, since the UML diagram formalism has a higher overall score.

Based on the output of the processing block 506, a class or formalismassignment 508 may be determined. In the above example, the inputstructure may be assigned to the UML Diagram formalism.

An input may contain structures from multiple different formalism types.Accordingly, different portions of the input may be analyzed separatelyto determine the multiple formalism types present. Furthermore, anoutput corresponding to the input may have multiple different formalismtypes, which may be embedded hierarchically and/or maintained separatelyfrom each other.

After an appropriate formalism type is selected, the input structure(s)corresponding to the formalism type may be processed using aformalism-specific technique. The same element type may be interpretedin different ways depending on the formalism assigned to the image. Forexample, one element type that may be present in an image is a fixedpoint formulation (e.g., an “fi” object in the MATLAB technicalcomputing environment), which specifies a particular representation of anumber using a slope and a bias. If the identified formalism supportsthe use of such a fixed point formulation, the input text identifyingthe slope and the bias may be properly interpreted so that the number iscorrectly represented in the output. The formalism-specific techniquemay support such translations. In some embodiments, certain programs,applications, environments, or processing engines may be associated withone or more formalisms. For example, block diagram models may be builtand executed using the Simulink® graphical modeling software by theMathWorks, Inc. of Natick Mass. Mathematical equations may berepresented in the MATLAB® technical computing environment (TCE), alsoby the MathWorks, Inc. Tables, charts, and plots may be designed in theMATLAB (TCE) or in the Excel® spreadsheet software by Microsoft Corp. ofRedmond, Wash. The formalism-specific technique may be applied by theprogram, application, environment, or engine associated with theformalism. Examples of such formalism-specific techniques are shown inFIGS. 6A-6C.

For example, FIG. 6A depicts a block-diagram-specific techniqueimplemented in Simulink modeling software. Using the model descriptionembodied by the input, a system may generate an equivalent Simulinkmodel that reflects the block locations, block sizes, blockdefinitions/types, and appropriate connections between blocks asrepresented in the input.

At step 610, a block and line detection algorithm may be run. The blockand line detection algorithm may analyze the input image to identifyrectangles (corresponding to blocks) and arrows (corresponding to lines)within the image. The algorithm may convert pixel coordinates of therectangles and arrows into a position for the blocks and lines in ablock diagram editor. At step 612, each of the blocks and lines detectedat step 610 may be analyzed to determine if the analyzed element is ablock or a line. For example, the processing may determine whether theanalyzed element occupies a certain minimum threshold area and has asubstantially four-sided shape generally corresponding to a rectangle.Elements satisfying these criteria may be classified as blocks.Otherwise, the element may be identified as a line.

If the outcome at step 612 is “no,” (i.e., the element is a line) thenprocessing may proceed to step 614, at which point a signal line may bedrawn at a location in the block diagram model corresponding to thelocation of the line in the input (and connecting the same elements aswere connected in the input). If one of the elements connected by theline has not yet been processed, then it may be processed at this timeto determine what type of element it is. Once the element type isidentified, the line may be made to connect to the identified element.

If the outcome at step 612 is “yes,” then processing may proceed to step616. When a block is detected, text written inside the identified blockmay be extracted. The system may perform optical character recognition(OCR) to recognize symbols in the text. If the OCR succeeds, then thesystem may use the OCR′ed text as the designated name of the block.

In case the OCR fails, the system may save a bitmap representation ofthe author's writing/diagram and insert it into the block diagram modelfor later review and updating.

At step 618, the system may use the text extracted at step 616 todetermine an identity of the block being drawn. For example, the systemmay compare the OCR′ed text to a list of names form a library of blocksassociated with the block diagram modeling environment. The system maydetermine whether the requested block exists within the library.

The system may consult a list of synonyms to determine an identity ofthe requested block, even if the requested block is not properlyidentified in the manner expected by the software. For example, a usermay designate a block using the text “ADD,” and there may not be an ADDblock in the library. However, the system may recognize that the word“ADD” as being synonymous with “SUM,” and may add a SUM block to themodel.

The system may also translate requested blocks into different (butequivalent) forms. For example, a user may request a “COS(X)” block,which the system may interpret as a cosine block operating on the value“x”. However, the block diagramming software may not implement a cosineblock, and the system may instead translate this request into theequivalent representation of “sin(x+pi/2),” which may then beimplemented using a sine block.

As an alternative (or in addition) to identifying blocks using text, asis done in steps 616-618, the system may use Simulink (or other blockdiagram software) specific block icons, or a custom dictionary, toidentify a type of the block.

If the answer at step 618 is “NO” (i.e., the requested block does notexist in the library), then processing may proceed to step 620. Ageneric block may be added to the block diagram model, and the genericblock may optionally reference a bitmap of the text from the input imagecorresponding to the block. Accordingly, implementation details maylater be added.

If the answer at step 622 is “YES,” (i.e., the requested block exists inthe library, then a block corresponding to the requested block may beretrieved from the library. The requested block may be added to themodel at the position indicated in the input. A size of the added blockmay correspond to a size of the block in the input.

It is noted that, although the procedure in FIG. 6A has been describedwith reference to block diagram models, this procedure may be readilyadapted to designing statecharts, as well as other formalisms andcombinations.

Another example of a formalism-specific technique is shown in FIG. 6B.Specifically, FIG. 6B depicts a table-specific technique implementedusing a MATLAB TCE.

At step 630, the system may identify an area corresponding to each cellarea in a table. For example, the system may recognize a number ofrectangles in the image, where each rectangle corresponds to a distinctcell. The area within the rectangles may be identified as a cell area.

At step 632, the system may run an optical character recognition (OCR)algorithm to extract characters and/or numbers from respective cellareas.

At step 634, the system may determine a data type in each cell based onan output of the OCR data. The characters and/or numbers may be examinedto identify whether the characters and/or numbers fit a predeterminedpattern that is mapped to a data type. For example, a series of numberswith no decimal point may be identified as an integer. A series ofnumbers with a decimal point may be identified as a double or floatingpoint number, as appropriate. A series of alphabetical characters alongwith zero or more numbers may be identified as a string.

At step 636, the system may construct a table data variable using aMATLAB TCE corresponding to each column (or row) of the table. The setsof characters/numbers identified at step 632 may be cast into the datatype identified at step 634, and stored in the table data variable. Thetable data variables corresponding to each column or row may be appendedtogether in order to create a table.

Yet another example of a formalism-specific technique is shown in FIG.6C. FIG. 6C depicts an equation-specific technique implemented, forexample, using a MATLAB TCE.

At step 640, the system may run an optical character recognition (OCR)algorithm to recognize symbols and characters in the equation. At step642, the system may determine whether all of the characters in theequation were able to be detected by the OCR algorithm. If some symbolsand/or characters are not detected processing may proceed to step 644.

At step 644, any symbols that were successfully detected may be analyzedin order to determine functions, products, toolboxes, classes, etc. thatmay relate to the equation. For example, if the input equation includedthe sum symbol, then the available functions and products may befiltered to remove any that do not include the sum symbol. Possiblematches to the requested equation may be displayed on a display device.A user may select one of the options to represent the equation.

Optionally, information about the determined functions, products,toolboxes, classes, etc. may be displayed on a display device. Forexample, help information specific to the determined functions,products, toolboxes, classes, etc. may be displayed. Information aboutthe formalism associated with the symbols, such as formalism-specifichelp information, may also be displayed.

Images of the unidentified characters may be saved for presentation on adisplay device alongside the related functions and products. This mayassist a user in evaluating the functions and products in order todetermine which function or product best matches the input.

If the answer at step 642 is “yes,” (i.e., all the symbols or characterswere successfully detected, then processing may proceed to step 646. Atstep 646, the system may change the recognized text of the equation intoan appropriate representation (such as LaTeX form, which allowsmathematical symbols to be accurately represented). Existing functionsor products available to the system may also represent equationscorresponding to the function or product in LaTeX form.

At step 648, LaTeX patterns in the equation may be used to search adatabase of existing products or functions. A product or function thatcorresponds to the LaTeX patterns in the equation may be selected as aprobable match for the equation. At step 650, code for the selectedLaTeX representation may be created using a suitable programmingenvironment, such as a Python environment, Octave, or the MATLAB TCE.For example, LaTeX commands may be identified and a mathematicalrepresentation corresponding to the LaTeX commands may be identified.These mathematical representations may also correspond to commands,identifiers, operations, etc., in the programming environment (e.g., aMATLAB command, identifier, or operation).

The above-described formalism-specific techniques may be used togenerate an output representation that corresponds to the inputstructure(s). In some embodiments, the corresponding outputrepresentation may be equivalent to the input structure(s). If aprogram, application, environment, or engine is associated with theformalism used to process the input, the program, application,environment, or engine may be used to represent, store, or process theoutput representation.

For example, FIG. 7 depicts an example of an input representation 700including a statechart, which is processed to generate a statechartusing the Stateflow graphical modeling environment 710.

Probability table 400 may be used to identify the input representation700 as corresponding to a statechart. For example, probability table 400may include elements 406 corresponding to states (e.g., circleelements), transitions (e.g., curved arrows), and text. If the text isinside of a circle, this may indicate that the circle represents a statein a statechart. If the text is outside of a circle and/or present nearone of the curved arrows, this may indicate that the text represents acondition for transitioning between two states. If a state is drawn astwo concentric circles, this may represent an initial state, which isrepresented in the statechart environment (in some embodiments) as anarrow with a solid circle as a source point (or other suitable syntax).

FIG. 7 depicts an output representation that directly corresponds to theinput. Alternatively (or in addition), the input structure(s) may betransformed or translated into a different format. Examples of suitabletransformations or translations are shown in FIGS. 8-12.

For example, FIG. 8 depicts an example of an equation 720. For example,the equation may originate in a textbook, and may be scanned by astudent. In order to identify the input as an equation, the probabilitytable 400 may include elements 406 corresponding to commonly-usedmathematical symbols and phrases (e.g., Σ, =, sin, etc.). The equationmay be translated using a formalism-specific technique such as the onedepicted in FIG. 6C.

The MATLAB (TCE) may perform optical character recognition and convertthe picture into a MATLAB equation 730. Alternatively or in addition,the MATLAB TCE may suggest a function and/or toolbox 735 that correspondto the written equation. The toolbox may include a number of functions,and the TCE may suggest the toolbox that includes an appropriatefunction or may suggest both the toolbox and the appropriate function.For example, the TCE may know that the control toolbox includes theappropriate function based on the context of the input, but the TCE maybe unable to determine the specific control toolbox function. In suchembodiments, the TCE may recommend the toolbox without specifying thespecific function to be used. Still further, the system may generate asolution to the equation and present the solution.

FIG. 9 depicts an example of a graphical chemical structure 740. Inorder to identify the image as including a chemical structure, theprobability table 400 may include elements 406 corresponding to chemicalbonds, element and molecule symbols, and common molecular structures(e.g., rings, commonly-used molecules which may be built into largermolecules).

The chemical structure may be analyzed in order to determine, forexample, a chemical formula for the structure. The chemical formula maybe compared to a database of chemical formulas to determine the name 750of the molecule or structure shown in the graphical chemical structure.

FIG. 10 depicts a UML diagram 760 describing classes (e.g., BankAccount,CheckingAccount, and SavingsAccount) that are each subclasses of theclass “Accounts.” The image of the UML diagram may be processed andconverted into a MATLAB class definition 770 implementing the class(es)described in the UML diagram.

In order to identify the input as including a UML diagram, theprobability table 400 may include elements 406 corresponding to arrows,multi-celled tables, and folders. Certain symbols in the text of theimage, such as “(”, “)”, and “:” may be given special significance andmay increase the probability that the image contains a UML diagram.

FIG. 11 depicts a table of data 780. The image of the table may beprocessed and converted into a graph 790. Alternatively or in addition,the table may be converted into MATLAB® table data.

For example, after recognizing that the table 780 belongs to the tableformalism, the system may examine the first row and first column toextract labels for the axes or lines of the graph. The system may alsoexamine predetermined locations (such as directly above or below thetable) to identify textual candidates that may represent a title for thegraph. Cells within the table may be used to generate data values to bepopulated in the graph at the locations designated by the row and columnheaders.

FIG. 12 depicts a still image extracted from a video. The still imagemay be processed in order to extract one or more structures (such as theblock diagram and/or the equation in the image) and may convert thosestructures into appropriate representations (such as a Simulink® modelof the block diagram, or a MATLAB® function corresponding to theequation).

The formalism-specific techniques may also include techniques relatingto physical domains, such as (but not limited to) electrical circuitdesigns (which may be translated into a circuit model), multi-bodysystems (which may be translated into a multibody diagram model or atime-based block diagram model), and CAD drawings (which may betranslated into, for example, a CAD representation in a CAD environment,or a model in a multibody modeling formalism such as SimMechanics™ ofThe MathWorks, Inc.). FIG. 13 depicts an example of a physicalformalism, in which a drawing of an electrical circuit is translatedinto a SimElectronics® model.

The formalism-specific techniques of FIGS. 6A-6C, and thetransformations/translations of FIGS. 7-12 are but a few exemplaryembodiments. Many other possibilities exist, depending on theapplication. Several additional suitable examples are described below:

-   -   An image of a tabular column of data may be provided to MATLAB®,        which may convert the column into a table variable. The table        variable may be added to an existing table.    -   A circuit designer may be provided with a handout showing an        electrical circuit, and may scan the handout. The resulting        scanned image may be provided to Simulink®, which may convert        the image into a SimElectronics® model.    -   A user may encounter a MATLAB® plot, and may want to know the        MATLAB® function that generated the plot. The user may take a        picture of the plot and provide the picture to MATLAB®. MATLAB®        may process the picture and suggest functions that would        generate a similar-looking plot. For example, MATLAB® may        identify relevant features of the input image that correspond to        MATLAB plot-generation functions, such as font sizes, axes        settings, legends, grid settings, colors, and markers.    -   A user may need to know what product or toolbox to use for a        particular task. The user may speak about the task to MATLAB®,        and MATLAB® may convert the audio description into a call to a        toolbox or product, and provide the described toolbox or product        to the user. For example, a speech-to-text utility may convert        spoken words into a format that can be processed and used by an        application, such as MATLAB.    -   An image of a time-based block diagram model may be scanned, and        code (such as C code, C++ code, HDL code, or PLC code) for the        time-based block diagram model may be output.    -   An image of source code may be scanned, the programming language        or languages of the source code may be identified and a data        structure of appropriate syntax in the identified programming        language or in another programming language may be generated.    -   A user may draw an input image of a time-based block diagram        model on a whiteboard. A camera may record the input image and a        processor may process the input image to transform the input        image into a Simulink® block diagram model. The translated block        diagram model may then be projected onto the whiteboard by a        projector, and the original drawing may be erased. The user may        then proceed to draw on the whiteboard to modify or annotate the        projected model.

One or more of the above-described acts may be encoded ascomputer-executable instructions executable by processing logic. Thecomputer-executable instructions may be stored on one or morenon-transitory computer readable media. One or more of the abovedescribed acts may be performed in a suitably-programmed electronicdevice. FIG. 14 depicts an example of an electronic device 800 that maybe suitable for use with one or more acts disclosed herein.

The electronic device 800 may take many forms, including but not limitedto a computer, workstation, server, network computer, Internetappliance, mobile device, a tablet computer, a smart sensor, applicationspecific processing device, etc.

The electronic device 800 is illustrative and may take other forms. Forexample, an alternative implementation of the electronic device 800 mayhave fewer components, more components, or components that are in aconfiguration that differs from the configuration of FIG. 14. Thecomponents of FIG. 14 and/or other figures described herein may beimplemented using hardware based logic, and/or logic that is acombination of hardware and software based logic (e.g., hybrid logic);therefore, components illustrated in FIG. 14 and/or other figures arenot limited to a specific type of logic.

The processor 810 may include hardware based logic or a combination ofhardware based logic and software to execute instructions on behalf ofthe electronic device 800. The processor 810 may include logic that mayinterpret, execute, and/or otherwise process information contained in,for example, the memory 820. The information may includecomputer-executable instructions and/or data that may implement one ormore embodiments of the invention. The processor 810 may comprise avariety of homogeneous or heterogeneous hardware. The hardware mayinclude, for example, some combination of one or more processors,microprocessors, field programmable gate arrays (FPGAs), applicationspecific instruction set processors (ASIPs), application specificintegrated circuits (ASICs), complex programmable logic devices (CPLDs),graphics processing units (GPUs), or other types of processing logicthat may interpret, execute, manipulate, and/or otherwise process theinformation. The processor may include a single core or multiple cores815. Moreover, the processor 810 may include a system-on-chip (SoC) orsystem-in-package (SiP). An example of a processor 810 is the Intel®Xeon® processor available from Intel Corporation, Santa Clara, Calif.

The electronic device 800 may include one or more tangiblenon-transitory computer-readable storage media for storing one or morecomputer-executable instructions that may implement one or moreembodiments of the invention. The non-transitory computer-readablestorage media may be, for example, the memory 820 or the storage 880.The memory 820 may comprise a random access memory (RAM) that mayinclude RAM devices that may store the information. The RAM devices maybe volatile or non-volatile and may include, for example, one or moreDRAM devices, flash memory devices, SRAM devices, zero-capacitor RAM(ZRAM) devices, twin transistor RAM (TTRAM) devices, read-only memory(ROM) devices, ferroelectric RAM (FeRAM) devices, magneto-resistive RAM(MRAM) devices, phase change memory RAM (PRAM) devices, or other typesof RAM devices.

The electronic device 800 may include a virtual machine (VM) 830 forexecuting instructions loaded in the memory 820. A virtual machine 830may be provided to handle a process running on multiple processors sothat the process may appear to be using only one computing resourcerather than multiple computing resources. Virtualization may be employedin the electronic device 800 so that infrastructure and resources in theelectronic device may be shared dynamically. Multiple VMs 830 may beresident on a single computing device 800.

A hardware accelerator 840, may be implemented in an ASIC, FPGA, or someother device. The hardware accelerator 840 may be used to reduce thegeneral processing time of the electronic device 800.

The electronic device 800 may include a network interface 850 tointerface to a Local Area Network (LAN), Wide Area Network (WAN) or theInternet through a variety of connections including, but not limited to,standard telephone lines, LAN or WAN links (e.g., T1, T3, 56 kb, X.25),broadband connections (e.g., integrated services digital network (ISDN),Frame Relay, asynchronous transfer mode (ATM), wireless connections(e.g., 802.11), high-speed interconnects (e.g., InfiniBand, gigabitEthernet, Myrinet) or some combination of any or all of the above. Thenetwork interface 850 may include a built-in network adapter, networkinterface card, personal computer memory card international association(PCMCIA) network card, card bus network adapter, wireless networkadapter, universal serial bus (USB) network adapter, modem or any otherdevice suitable for interfacing the electronic device 800 to any type ofnetwork capable of communication and performing the operations describedherein.

The electronic device 800 may include one or more input devices 860,such as position-based input devices, velocity-based input devices,visual input devices, spatial input devices, audio input devices,wearable input devices, stationary and mobile input devices, augmentedreality devices, and projections. Examples of such input devicesinclude, but are not limited to a keyboard, an interactive whiteboard, amulti-point touch interface, a pointing device (e.g., a mouse), agyroscope, an accelerometer, a haptic device, a tactile device, a neuraldevice, a microphone, or a camera that may be used to receive inputfrom, for example, a user. Note that electronic device 800 may includeother suitable I/O peripherals.

The input devices 860 may allow a user to provide input. The input maybe registered on a visual display device 870. A graphical user interface(GUI) 875 may be shown on the display device 870.

A storage device 880 may also be associated with the electronic device800. The storage device 880 may be accessible to the processor 810 viaan I/O bus or other suitable mechanism and/or technique. The informationin the storage device 880 may be executed, interpreted, manipulated,and/or otherwise processed by the processor 810. The storage device 880may include, for example, a magnetic disk, optical disk (e.g., CD-ROM,DVD player), random-access memory (RAM) disk, tape unit, and/or flashdrive. The information may be stored on one or more non-transienttangible computer-readable media contained in the storage device. Thismedia may include, for example, magnetic discs, optical discs, magnetictape, and/or memory devices (e.g., flash memory devices, static RAM(SRAM) devices, dynamic RAM (DRAM) devices, or other memory devices).The information may include data and/or computer-executable instructionsthat may implement one or more embodiments of the invention

The storage device 880 may be used for storing one or more files 882,such as the above-described probability table 400. The storage device880 may further store applications 884, such as the above-describedprograms, applications, environments, or engines.

The electronic device 810 can further be running an operating system(OS) 886. Examples of OS 886 may include the Microsoft® Windows®operating systems, the Unix and Linux operating systems, the MacOS® forMacintosh computers, an embedded operating system, such as the SymbianOS, a real-time operating system, an open source operating system, aproprietary operating system, operating systems for mobile electronicdevices, or other operating system capable of running on the electronicdevice and performing the operations described herein. The operatingsystem may be running in native mode or emulated mode.

Additionally, the storage device 880 may store logic corresponding tothe pre-processing logic 300, the classification model 500, andfeature-specific processing logic 890.

One or more embodiments of the invention may be implemented usingcomputer-executable instructions and/or data that may be embodied on oneor more non-transitory tangible computer-readable mediums. The mediumsmay be, but are not limited to, a hard disk, a compact disc, a digitalversatile disc, a flash memory card, a Programmable Read Only Memory(PROM), a Random Access Memory (RAM), a Read Only Memory (ROM),Magnetoresistive Random Access Memory (MRAM), a magnetic tape, or othercomputer-readable media.

One or more embodiments of the invention may be implemented in aprogramming language. Some examples of languages that may be usedinclude, but are not limited to, Python, C, C++, C#, SystemC, Java,Javascript, a hardware description language (HDL), unified modelinglanguage (UML), and Programmable Logic Controller (PLC) languages.Further, one or more embodiments of the invention may be implemented ina hardware description language or other language that may allowprescribing computation. One or more embodiments of the invention may bestored on or in one or more mediums as object code. Instructions thatmay implement one or more embodiments of the invention may be executedby one or more processors. Portions of the invention may be ininstructions that execute on one or more hardware components other thana processor.

In some embodiments, the electronic device 800 may be part of a networkimplementation. FIG. 15 depicts an exemplary network implementation thatmay implement one or more embodiments of the invention. A system 900 mayinclude an electronic device 800, a network 910, a service provider 920,a target environment 930, and a cluster 940. The embodiment of FIG. 15is exemplary, and other embodiments can include more devices, fewerdevices, or devices in arrangements that differ from the arrangement ofFIG. 9.

The network 910 may transport data from a source to a destination.Embodiments of the network 910 may use network devices, such as routers,switches, firewalls, and/or servers (not shown) and connections (e.g.,links) to transport data. Data may refer to any type of machine-readableinformation having substantially any format that may be adapted for usein one or more networks and/or with one or more devices (e.g., theelectronic device 800, the service provider 920, etc.). Data may includedigital information or analog information. Data may further bepacketized and/or non-packetized.

The network 910 may be a hardwired network using wired conductors and/oroptical fibers and/or may be a wireless network using free-spaceoptical, radio frequency (RF), and/or acoustic transmission paths. Inone implementation, the network 910 may be a substantially open publicnetwork, such as the Internet. In another implementation, the network910 may be a more restricted network, such as a corporate virtualnetwork. The network 910 may be the Internet, an intranet, Local AreaNetwork (LAN), Wide Area Network (WAN), Metropolitan Area Network (MAN),wireless network (e.g., using IEEE 802.11), or other types of network.The network 910 may use middleware, such as Common Object Request BrokerArchitecture (CORBA) or Distributed Component Object Model (DCOM)Implementations of networks and/or devices operating on networksdescribed herein are not limited to, for example, any particular datatype, protocol, and/or architecture/configuration.

The service provider 920 may include a device that makes a serviceavailable to another device. For example, the service provider 920 mayinclude an entity (e.g., an individual, a corporation, an educationalinstitution, a government agency, etc.) that provides one or moreservices to a destination using a server and/or other devices. Servicesmay include instructions that are executed by a destination to performan operation (e.g., an optimization operation). Alternatively, a servicemay include instructions that are executed on behalf of a destination toperform an operation on the destination's behalf.

The target environment 930 may include a device that receivesinformation over the network 910. For example, the target environment930 may be a device that receives user input from the electronic device900.

The cluster 940 may include a number of units of execution (UEs) 950 andmay perform processing on behalf of the electronic device 800 and/oranother device, such as the service provider 920. For example, thecluster 940 may perform parallel processing on an operation receivedfrom the electronic device 800. The cluster 940 may include UEs 950 thatreside on a single device or chip or that reside on a number of devicesor chips.

The units of execution (UEs) 950 may include processing devices thatperform operations on behalf of a device, such as a requesting device. AUE may be a microprocessor, field programmable gate array (FPGA), and/oranother type of processing device. UE 950 may include code, such as codefor an operating environment. For example, a UE may run a portion of anoperating environment that pertains to parallel processing activities.The service provider 920 may operate the cluster 940 and may provideinteractive optimization capabilities to the electronic device 800 on asubscription basis (e.g., via a web service).

Units of Execution (UEs) may provide remote/distributed processingcapabilities for products such as the MATLAB TCE. A hardware unit ofexecution may include a device (e.g., a hardware resource) that mayperform and/or participate in parallel programming activities. Forexample, a hardware unit of execution may perform and/or participate inparallel programming activities in response to a request and/or a taskit has received (e.g., received directly or via a proxy). A hardwareunit of execution may perform and/or participate in substantially anytype of parallel programming (e.g., task, data, stream processing, etc.)using one or more devices. For example, a hardware unit of execution mayinclude a single processing device that includes multiple cores or anumber of processors. A hardware unit of execution may also be aprogrammable device, such as a field programmable gate array (FPGA), anapplication specific integrated circuit (ASIC), a digital signalprocessor (DSP), or other programmable device. Devices used in ahardware unit of execution may be arranged in many differentconfigurations (or topologies), such as a grid, ring, star, or otherconfiguration. A hardware unit of execution may support one or morethreads (or processes) when performing processing operations.

A software unit of execution may include a software resource (e.g., atechnical computing environment) that may perform and/or participate inone or more parallel programming activities. A software unit ofexecution may perform and/or participate in one or more parallelprogramming activities in response to a receipt of a program and/or oneor more portions of the program. A software unit of execution mayperform and/or participate in different types of parallel programmingusing one or more hardware units of execution. A software unit ofexecution may support one or more threads and/or processes whenperforming processing operations.

The term ‘parallel programming’ may be understood to include multipletypes of parallel programming, e.g. task parallel programming, dataparallel programming, and stream parallel programming Parallelprogramming may include various types of processing that may bedistributed across multiple resources (e.g., software units ofexecution, hardware units of execution, processors, microprocessors,clusters, labs) and may be performed at the same time.

For example, parallel programming may include task parallel programmingwhere a number of tasks may be processed at the same time on a number ofsoftware units of execution. In task parallel programming, a task may beprocessed independently of other tasks executing, for example, at thesame time.

Parallel programming may include data parallel programming, where data(e.g., a data set) may be parsed into a number of portions that may beexecuted in parallel using, for example, software units of execution. Indata parallel programming, the software units of execution and/or thedata portions may communicate with each other as processing progresses.

Parallel programming may include stream parallel programming (sometimesreferred to as pipeline parallel programming) Stream parallelprogramming may use a number of software units of execution arranged,for example, in series (e.g., a line) where a first software unit ofexecution may produce a first result that may be fed to a secondsoftware unit of execution that may produce a second result given thefirst result. Stream parallel programming may also include a state wheretask allocation may be expressed in a directed acyclic graph (DAG) or acyclic graph.

Other parallel programming techniques may involve some combination oftask, data, and/or stream parallel programming techniques alone or withother types of processing techniques to form hybrid-parallel programmingtechniques.

The foregoing description may provide illustration and description ofvarious embodiments of the invention, but is not intended to beexhaustive or to limit the invention to the precise form disclosed.Modifications and variations may be possible in light of the aboveteachings or may be acquired from practice of the invention. Forexample, while a series of acts has been described above, the order ofthe acts may be modified in other implementations consistent with theprinciples of the invention. Further, non-dependent acts may beperformed in parallel.

In addition, one or more implementations consistent with principles ofthe invention may be implemented using one or more devices and/orconfigurations other than those illustrated in the Figures and describedin the Specification without departing from the spirit of the invention.One or more devices and/or components may be added and/or removed fromthe implementations of the figures depending on specific deploymentsand/or applications. Also, one or more disclosed implementations may notbe limited to a specific combination of hardware.

Furthermore, certain portions of the invention may be implemented aslogic that may perform one or more functions. This logic may includehardware, such as hardwired logic, an application-specific integratedcircuit, a field programmable gate array, a microprocessor, software, ora combination of hardware and software.

No element, act, or instruction used in the description of the inventionshould be construed critical or essential to the invention unlessexplicitly described as such. Also, as used herein, the article “a” isintended to include one or more items. Where only one item is intended,the term “a single” or similar language is used. Further, the phrase“based on,” as used herein is intended to mean “based, at least in part,on” unless explicitly stated otherwise. In addition, the term “user”, asused herein, is intended to be broadly interpreted to include, forexample, an electronic device (e.g., a workstation) or a user of aelectronic device, unless otherwise stated.

It is intended that the invention not be limited to the particularembodiments disclosed above, but that the invention will include any andall particular embodiments and equivalents falling within the scope ofthe following appended claims.

1. A non-transitory computer readable medium storing instructions, theinstructions comprising: one or more instructions that, when executed,cause a processor to: receive an input, the input comprising a pluralityof input elements; parse the input to identify: a first element fromamong the plurality of input elements, the first element representedaccording to a formalism type that identifies a syntax for organizingelements of the formalism type, and a second element from among theplurality of input elements, the second element represented according tothe formalism type; access a library comprising entries for differentformalism types, the formalism type included in the different formalismtypes; determine a likelihood that the first element and the secondelement coexist together in a formalism type from among the differenttypes of formalisms in the library, and select, based on thedetermining, a selected formalism type that is consistent with acoexistence of the first element and the second element; and generate anoutput, the output: comprising the first element and the second element,and at least partly represented in a format corresponding to theselected formalism type.
 2. The medium of claim 1, wherein the inputcomprises multiple formalism types such that the plurality of inputelements of the input include a first set of elements representedaccording to a first formalism type and a second set of elementsrepresented according to a second formalism type distinct from the firstformalism type.
 3. The medium of claim 1, wherein the output comprisesoutput elements corresponding to a plurality of distinct formalisms suchthat the output is partly represented in the format corresponding to theselected formalism type, and is further represented in a second formatcorresponding to a second selected formalism type that is distinct fromthe selected formalism type.
 4. The medium of claim 1, wherein thedifferent formalism types represent one or more of: a table, atime-based block diagram model, a statechart, a Unified Model Languagediagram, a mathematical equation, a chemical structure, a circuit model,a multibody model a chart, and a plot
 5. The medium of claim 1, whereindifferent formalism types are processed by different types of processingengines or environments.
 6. The medium of claim 1, wherein differentformalism types have different alphabets, syntaxes, or semantics.
 7. Themedium of claim 1, wherein the input comprises one or more of: an image;video; audio; and gesture data.
 8. The medium of claim 1, wherein thelibrary comprises a table of probabilities that a type of element isfound in each of the formalism types in the library.
 9. The medium ofclaim 8, wherein the probabilities are dynamically updated based onuser-feedback about a classification of the input into a formalism type.10. The medium of claim 8, wherein the probabilities are dynamicallyupdated based on historical classifications of formalism types.
 11. Themedium of claim 8, wherein analyzing the first element and the secondelement comprises: determining a first probability, the firstprobability representing a probability that the first element belongs tothe selected formalism type; determining a second probability, thesecond probability representing a probability that the second elementbelongs to the selected formalism type; and adding the first probabilityto the second probability.
 12. The medium of claim 1, wherein the secondelement is identified, at least in part, based on a predicted formalismtype that is determined based on an analysis of the first element. 13.The medium of claim 1, wherein generating the output comprises applyinga formalism-specific rule to the first element or the second element,the formalism-specific rule being particular to the selected formalismtype.
 14. The medium of claim 1, wherein generating the output comprisesprocessing the first element and the second element using aformalism-specific procedure.
 15. The medium of claim 1, furthercomprising displaying help information specific to the selectedformalism type, the first element, or the second element.
 16. A methodcomprising: receiving, using a processor of a computing device, aninput, the input comprising a plurality of input elements; parsing theinput, the parsing: performed using the processor, identifying a firstelement from among the plurality of input elements, and identifying asecond element from among the plurality of input elements, wherein thefirst element and the second element are represented according to aformalism type defining a syntax for organizing elements of theformalism type; access a library comprising entries for differentformalism types; evaluating a likelihood that the first element and thesecond element coexist together in a formalism type from among thedifferent formalism types in the library, and selecting, based on theevaluating, a selected formalism type that is consistent with acoexistence of the first element and the second element; and generating,using the processor, an output, the output: representing a translationof at least part of the input into an output type that is associatedwith the selected formalism type.
 17. The method of claim 16, whereinthe selected formalism type is an equation, and the translation is asolution to the equation.
 18. The method of claim 16, wherein theselected formalism type is an equation, and the translation is a sourcecode function representing the equation.
 19. The method of claim 16,wherein the selected formalism type is a state diagram, and thetranslation is a statechart model representing the state diagram. 20.The method of claim 16, wherein the selected formalism type is a plot,and the translation is a source code command that, when executed,generates the plot.
 21. The method of claim 16, wherein either: theselected formalism type is a table, and the translation is a plot orchart representing the table, or the selected formalism type is a plotor chart, and the translation is a table of data represented in theplot.
 22. The method of claim 16, wherein the selected formalism type isa Unified Model Language (UML) diagram, and the translation is a sourcecode class representing the UML diagram.
 23. The method of claim 16,wherein the selected formalism type is a chemical formula, and thetranslation is a name of a chemical represented by the chemical formula.24. The method of claim 16, wherein either: the selected formalism typeis a block diagram model, and the translation is a source coderepresentation of the block diagram model, or the selected formalismtype represents source code, and the translation is a block diagrammodel representing the source code.
 25. A system comprising: anon-transitory computer readable medium storing an input, the input:comprising a plurality of input elements, and being uninterpretable byan engine associated with a selected formalism type; and a processorprogrammed with instructions that, when executed, cause the processorto: parse the input to identify: a first element from among theplurality of input elements, and a second element from among theplurality of input elements, wherein the first element and the secondelement are represented according to a formalism type defining a syntaxfor organizing elements of the formalism type; access a librarycomprising entries for different formalism types, the differentformalism types comprising the selected formalism type associated withthe engine; analyze the first element and the second element, theanalyzing: examining the first element in context with the secondelement to evaluate a likelihood that the first element and the secondelement coexist together in the selected formalism type, and selecting,based on the examining, the selected formalism that is consistent with acoexistence of the first element and the second element; and generate anoutput, the output: comprising the first element and the second element,and at least partly represented in a format corresponding to theselected formalism type, wherein the output is interpretable by theengine associated with the selected formalism.